[FLINK-39958] Honor per-deployment Flink REST client timeout on rescale - #33
Closed
pobed2 wants to merge 1 commit into
Closed
[FLINK-39958] Honor per-deployment Flink REST client timeout on rescale#33pobed2 wants to merge 1 commit into
pobed2 wants to merge 1 commit into
Conversation
Autoscaler in-place rescales intermittently fail with a bare TimeoutException out of NativeFlinkService.updateVertexResources. When that happens the operator abandons in-place scaling with no retry and falls back to a full savepoint-and-redeploy: suspend with savepoint, delete the JobManager deployment, delete HA metadata, redeploy. On the webhooks-production-java-change-event-producer FlinkDeployment that costs ~150s of no checkpoint progress plus a slow-recovery tail, and it fired 7 times in 24h on that pipeline alone (17 times fleet-wide across three pipelines) in place of what is normally a ~3s rescale. The budget for that REST call is kubernetes.operator.flink.client.timeout (default 10s), which is operator-global: it is read from FlinkOperatorConfiguration, built by FlinkConfigManager.getDefaultConfig from the operator's own configuration, so a pipeline owner cannot raise it from their FlinkDeployment spec. The per-deployment-looking job.autoscaler.flink.rest-client.timeout is silently clobbered, which is FLINK-39958. Two changes: - Cherry-pick FLINK-39958 (apache#1141) so FlinkResourceContext only seeds AutoScalerOptions.FLINK_CLIENT_TIMEOUT from the operator timeout when the deployment has not set it. - Apply the same honor-the-per-deployment-value rule to the in-place rescale REST calls in NativeFlinkService. Those read operatorConfig directly and so are untouched by FLINK-39958, which is why the cherry-pick alone would not stop the teardowns. No defaults change. With job.autoscaler.flink.rest-client.timeout unset, both paths resolve to the operator-global timeout exactly as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Pierre-Olivier Bédard <pierre.bedard@shopify.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue: https://issues.apache.org/jira/browse/FLINK-39958
Requested by: Pierre-Olivier Bédard pierre.bedard@shopify.com
Slack thread: https://shopify.slack.com/archives/C0AKLLE8GPJ/p1785358236166909
What
FlinkResourceContextnow only seedsAutoScalerOptions.FLINK_CLIENT_TIMEOUTfrom the operator-global timeout when the deployment has not set it.NativeFlinkService, which readoperatorConfigdirectly and are therefore untouched by FLINK-39958.job.autoscaler.flink.rest-client.timeoutunset, both paths resolve to the operator-global timeout exactly as before.Why
kubernetes.operator.flink.client.timeout(default 10s) is the budget for thePUT /jobs/<id>/resource-requirementscall the operator makes against a running JobManager. It is operator-global — read fromFlinkOperatorConfiguration, built byFlinkConfigManager.getDefaultConfigfrom the operator's own configuration — so a pipeline owner cannot raise it from theirFlinkDeploymentspec, and the per-deployment-lookingjob.autoscaler.flink.rest-client.timeoutis silently clobbered.When that call times out, the operator abandons in-place scaling with no retry and falls back to a full savepoint-and-redeploy: suspend with savepoint, delete the JobManager deployment, delete HA metadata, redeploy. On
webhooks-production-java-change-event-producerthat turns a ~3s rescale into ~150s of no checkpoint progress plus a slow-recovery tail. It fired 7 times in 24h on that pipeline, 17 times fleet-wide across three pipelines.Raising the tolerance is the lever rather than more JobManager capacity: JM process CPU sat at 7-18% of its one core throughout, and the call blocks behind the adaptive scheduler's main thread while it rebuilds the execution graph and registers newly requested TaskManagers.
Details
getVertexResourcesas well asupdateVertexResources. Both are REST calls inside the samescale()try block, both were pinned to the operator-global timeout, and both surface as the sameError while rescaling, falling back to regular upgrade.getVertexResourcesruns first, so leaving it at 10s would keep the teardown path open. Happy to narrow this toupdateVertexResourcesonly if you prefer the smaller surface.mvn verify -pl flink-kubernetes-operator: 2113 tests pass, 0 Checkstyle violations, spotless clean.